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The origin of the monoclinic distortion and domain formation in the quasi two-dimensional layer 
compound NbTe2 is investigated. Angle-resolved photoemission shows that the Fermi surface is 
pseudogapped over large portions of the Brillouin zone. A b initio calculation of the electron and 
phonon bandstructure as well as the static RPA susceptibility lead us to conclude that Fermi surface 
nesting and electron-phonon coupling play a key role in the lowering of the crystal symmetry and 
in the formation of the charge density wave phase. 
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I. INTRODUCTION 



NbTe2 belongs to the category of layered transition 
metal dichalcogenides (TMDC) known for their quasi 
two-dimensional (2D) properties. Due to the reduced 
dimensionality, free charge carriers and phonons are cou- 
pling in an unique fashion, leading to the formation of 
charge density waves (CDW) and superconductivity. 
The competition between these two electronic ground- 
states is especially interesting in view of the anomalous 
properties of another class of strongly anisotropic lay- 
ered materials, the high T c cuprate superconductors. At- 
tempts to explain the complexity of electronic and mag- 
netic properties observed in these compounds are based 
on the subtle balancing of competing interactions pro- 
ducing superconducting pairing, spin and charge order- 
ing. For NbTe2, magnetic degrees of freedom are unlikely 
to play an important role. Thus, in principle NbTe2 al- 
lows to isolate the effects associated with density wave 
instabilities and superconductivity. 

An explanation for the CDW transition in 2D materials 
is derived from the theory for the Peierls instability Q 
in ID metals. A system of conduction electrons may un- 
der suitable conditions become unstable with respect to 
a spatially modulated perturbation with wavevector q, 
such as a static periodic lattice distortion. Kohn Q has 
shown that such soft phonon modes may result from the 
screening of lattice vibrations by conduction electrons. 
According to linear response theory, the quality of the 
screening by the electrons is measured by the static gen- 
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eralized susceptibility with Fourier component x(q). In- 
stability sets in when this quantity diverges. This hap- 
pens under favorable nesting conditions for which large 
portions of the Fermi surface can be connected or nested 
by a single q vector. Even when the system is not truly 
one-dimensional, nesting may become important, if the 
Fermi surface consists of flat parallel sheets. However 2D 
systems often remain metallic, since the opening of the 
gap removes only parts of the Fermi surface. 
The distortion already observed at room temperature in 
NbTe2 and the isostructural TaTe2 suggests the action of 
a single-axis CDW. The structure 3] is a monoclinically 
deformed version of the trigonal IT poly type, in which 
the transition metal sits in octahedrally coordinated sites 
between the chalcogen atoms (see Appendix D for struc- 
tural details). The metal atoms are displaced from the 
center of the coordination unit and the chalcogen layers 
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FIG. 1: Side view of the monoclinic structure of NbTe2 (pro- 
jection onto the (010) plane within the monoclinic space 
group). Each layer of NbTe2 consists of a Te-Nb-Te sand- 
wich. The Nb atoms (filled circles) are displaced within the 
plane and form 'trimers', whereas the Te atoms (empty cir- 
cles) exhibit an out-of-plane buckling (a,b,c). Successive Te- 
Nb-Te sandwiches are shifted within the plane. The stacking 
sequence is repeated after 3 layers. 



2 



form zig-zag chains to accomodate these shifts (see Fig. 
[J). After cooling of heat pulsed crystals to room tem- 
perature, transmission electron diffraction experiments 
revealed a second, seemingly unrelated triple-axis CDW 
state with a (>/l9 x \/l9) signature commensu- 
rate at room temperature, but readily rendered incom- 
mensurate just above. 

NbTe2 and TaTe2 are semimetals Q- The resistiv- 
ity decreases monotonically with decreasing temperature 
EJ El- A drop in resistivity of NbTe2 in the range 
0.5 — 0.74 K marks the transition into the superconduct- 
ing phase . Superconductivity is absent in TaTe2 . 
After a discussion of the experimental and theoretical 
methods, we will investigate the interplay between elec- 
tronic and structural properties of NbTe2- Our low en- 
ergy electron diffraction (LEED) measurements will al- 
low an alternative parametrization of the structure pro- 
posed by X-ray diffraction measurements, which is more 
appropriate in the discussion of electron dynamics. Nu- 
merous studies of the electronic properties have been car- 
ried out for disulfides HQ[m[ll[il[il[ill230 
and diselenides [T(IIT&l22L l23ll24|. Direct measurements 
of the Fermi surface topology of NbTe2 and TaTe2 have 
never been reported. In order to shed light on the ori- 
gin of the CDW phase and the domain formation ob- 
served in these compounds, we have measured the Fermi 
surface via full-hemispherical angle-resolved photoelec- 
tron spectroscopy (ARPES). Scanning tunneling spec- 
tropscopy (STS) experiments complement the ARPES 
data. We suggest that the distortion and the accom- 
panying domain formation is intimately related to the 
Fermi surface topology. Quantitative assessments of its 
nesting tendencies are obtained from first principle band- 
structure calculations. A computation of the vibrational 
spectrum and a soft mode analysis support our conclu- 
sions. 

In the following, we concentrate on NbTe2- Similarities 
and differences between NbTe2 and TaTe2, especially the 
absence of superconductivity in TaTe2, will be addressed 
at the end. 



II. EXPERIMENT AND CALCULATION 

Full hemispherical ARPES experiments were per- 
formed in a modified Vacuum Generator ESCALAB 
Mark II spectrometer with a residual gas pressure of 
2 x 10" 11 mbar equipped with a Mg K a (hv = 1253.6 
eV) X-ray anode, a monochromatized He discharge lamp 
providing He la (hv = 21.2 eV) radiation 25], and 
a three channeltron hemispherical electrostatic analyzer 
kept fixed in space during measurements. The sample is 
mounted on a manipulator with two motorized and com- 
puter controlled rotational axes and may be cooled via 
a closed cycle refrigerator. Energy resolution is 50 meV, 
the combined angular resolution of sample manipulator 
and analyzer is approximately 1°. Surface cleanness and 
quality before and after ARPES measurements was mon- 




FIG. 2: a) LEED pattern (75 eV) of NbTe 2 at room temper- 
ature, b) Schematic illustration of a single domain (3x1) 
superstructure as observed in reciprocal space. The fat dots 
indicate the T points of the undistorted trigonal lattice. Due 
to the (3 x 1) distortion in real space, two additional spots 
appear in between these main reflections represented by the 
small dots. Their spacing is |a*. The bold hexagons out- 
line the surface Brillouin zone of the trigonal (lxl) struc- 
ture. The thin compressed hexagons show the surface Bril- 
louin zone of the (3 x 1) superlattice. c) Projection onto the 
basal plane of the first three Te-Nb-Te sandwiches of the mon- 
oclinic structure. The size of the circles is proportional to the 
coordinate. The lattice vectors of the (lxl) and (3x1) 
cells are indicated, d) Bulk and surface Brillouin zone for 
1 T-NbTe2 with high symmetry points. 



itored by X-ray photoelectron spectroscopy (XPS) and 
checked with low energy electron diffraction (LEED) re- 
spectively. Orientation of the sample was achieved by 
X-ray photoelectron diffraction (XPD) IHEzl- 
Scanning tunneling microscopy (STM) and spectroscopy 
(STS) experiments were carried out with Pt/Ir tips using 
an Omicron LT-STM in a separate UHV system with a 
base pressure of 3 x 10 -11 mbar. Pure NbTe2 samples 
were prepared by the standard flux growing techniques. 
Sample cleavage was carried out in UHV using adhesive 
tape. 

First principle calculations were performed in the frame- 
work of density functional theory (DFT) using the 
full potential augmented plane wave plus local orbitals 
(APW+lo) method in conjunction with the generalized 
gradient approximation (GGA) in the parametrization 
of Perdew, Burke and Ernzerhof [28] as implemented 
in the WIEN2k software package [29| as well as the 
ABINIT code [3Q, |3l| using the local density approxima- 
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tion (LDA) and relativistic separable dual-space Gaus- 
sian pseudopotentials 32] for both Nb and Te, taking 
into account the Nb semicore states. 
A recent extension to WIEN2k based on the OPTICS 
package allows the computation of the frequency depen- 
dent random phase approximation (RPA) susceptibility 
|33|. The phonon dispersion is computed with the help 
of the linear response or density functional perturbation 
theory (DFPT) capabilities of ABINIT Hi^. Compu- 
tational details are given in [36^ . 



III. RESULTS AND DISCUSSION 

A. (3 x 1) surface superstructure 

Transmission electron microscope (TEM) images of 
NbTe2 show a complicated domain structure (see Q). 
Although the crystal structure within the domains is 
known, insights into the domain structure can be ob- 
tained from LEED experiments. Figure EK) presents a 
LEED measurement taken with electrons accelerated to a 
kinetic energy of 75.0 eV. An interpretation based on the 
monoclinic reciprocal lattice is not straightforward. The 
experimental pattern is most easily understood by con- 
sidering the undistorted trigonal parent structure. Since 
LEED patterns exhibit Bragg reflections of the 2D sur- 
face lattice, the surface Brillouin zone borders are super- 
imposed. The bold hexagons correspond to the surface 
Brillouin zone of the trigonal (lxl) structure. Due to 
the monoclinic distortion of the lattice, two additional 
spots in between the main reflections are visible, which 
can be understood in terms of a (3 x 1) superstructure. A 
schematic illustration is given in Fig. ^p). The presence 
of this superstructure results in a new surface Brillouin 
zone shown in Fig. ^jp). The LEED pattern results from 
the superposition of three orientational variants, rotated 
by 120° with respect to eachother, of such surface su- 
perstructures. Note that a (3 x 3) superstructure would 
result in additional spots occurring in the center of the 
triangle outlined by the white dots in Fig. A close 
inspection of the monoclinic structure in Fig. Et) reveals 
a (3 x 1) surface unit cell, confirming the experimental 
finding. The presence of additional spots were already 
reported earlier |33 and correctly interpreted as a super- 
position of patterns from three domains. However, the 
appearance of these superspots has not been recognized 
as a (3 x 1) surface superstructure. For completeness, we 
note that the bulk structure exhibits a (3 x 1 x 3) super- 
cell structure, since successive layers are shifted within 
the plane (see Fig. [IJ. 

The diffuse nature of the LEED pattern has been ex- 
plained in terms of thermal disorder 37]. Our LEED 
measurements at low temperature remain diffuse. The 
presence of a fine domain structure might be responsible 
for the broadening of the reflections. The average do- 
main size is obtained by comparison of the width of the 
reflections with the inter-reflection distance. From the 
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FIG. 3: a) Angular distribution of electrons from Ef mapped 
as a function of k|| at room temperature, white and black 
correspond to high and low intensity, respectively, b) Sym- 
metrized and flattened data of a), c) Theoretical APW+lo 
Fermi surface contours for the undistorted trigonal NbTe2 in 
the free electron final state approximation [l^| with photon 
energy hv — 21.2 eV, workfunction cj) = 4 eV and an assumed 
inner potential Vb = 13 eV. Brillouin zones of the (1 x 1) and 
one particular orientation of the (3x1) lattice are superim- 
posed. 



measurement in Fig. Et) with an average peak width of 
0.2 A" 1 , we estimate the average domain size to be of 
the order of 32 A, which agrees quite well with the do- 
main size from a sample from the same batch observed 
by STM (see below). 

LEED is not able to distinguish between three or six ori- 
entational variants of the (3x 1) superstructure. Whereas 
LEED probes the periodicity of the surface, XPD indi- 
cates the symmetry of the local environment of the emit- 
ting atom. The Te 3^/2 XPD diffractogram (not shown) 
exhibits a three-fold symmetry, which clearly shows that 
only three and not six orientational variants are present, 
since the presence of domains rotated by 180° would re- 
sult in a six-fold symmetry. We observed further that 
XPD diffractograms from different cleavage planes were 
rotated by 180° with respect to eachother, retaining how- 
ever their three-fold symmetry, indicating a change in the 
stacking sequence between successive Te-Nb-Te layers. 
The presence of the (3x1) superstructure implies a recon- 
struction of the Brillouin zone. In ARPES experiments, 
we may thus expect to observe the opening of a gap at the 
new Brillouin zone border accompanied by a backfolding 
of bands. 



B. Fermi surface topology 

Figure [3| presents Fermi surface maps (FSM) of NbTe2 
measured at room temperature, i.e. the intensity distri- 
bution for electrons from the Fermi level (Ef). These 
maps bear in many respects a close resemblance to the 
FSM's of isopolytopic TaS 2 and TaSe 2 [13, although in 
these materials, the CDW distortion is of the (y/T3xy/l3) 
type. Figure |3Ji) gives experimental raw data without 
any further treatment. Near normal emission, high in- 
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tensity is measured which falls off rather quickly towards 
larger polar angles. A similar behavior is observed for 
7T-TaS2 and 7T-TaSe2 and has been attributed to the 
d z 2 character of the transition metal band. A normal- 
ization of the FSM by the mean intensity for each polar 
emission angle as shown in Fig. |3t>) eliminates this de- 
pendence and allows to reveal weaker off- normal features. 
Centered circular features are then suppressed. 
The washed out character of the experimental FSM con- 
tours is another common feature of the 1 T family. Since 
the width in k|| of the bands is independent of temper- 
ature, we discharge thermally populated phonons as the 
origin of the broadening. From the monoclinic distor- 
tion it is expected, that the d z i band, which is mainly 
responsible for the spectral weight observed at the Fermi 
energy, splits into several subbands. This results in a 
larger width of the observed band in energy and conse- 
quently also in momentum. Secondly, in the presence 
of the domain structure, the coherence length of Bloch 
electrons must be of the order of the average domain size, 
since electrons get scattered at the domain boundaries. 
As in LEED, this should lead to a broadening of the 
crystal momentum of about 0.2 A -1 , consistent with the 
broadening observed in the experimental FSM. 
Great care is required in determining the Fermi sur- 
face crossings from the experimental data. Due to the 
weak dispersion and k-dependent photoemission matrix 
elements, which lead to intensity variations which have 
nothing to do with Fermi crossings, an unambiguous ex- 
traction of the Fermi surface needs additional informa- 
tion. We have measured the energy dispersion of the 
NbTe 2 d z 2 band along fM (Fig. Ha)), fM' and fK at 
room temperature and at T<20 K (not shown). The 
band topology is not affected by this change in tempera- 
ture indicating the absence of a phase transition in this 
temperature range. In order to determine the location of 
Fermi crossings, we applied the symmetrization method 
described in Ref. 38] . No quasiparticle crossing has been 
found for any of the measured energy dispersion curves. 
Thus, strictly speaking, the maps in Fig. [3|are not Fermi 
surfaces. The observed intensity originates from bands 
which come close to the Fermi level, but must not be as- 
sociated with quasiparticle crossings, but rather spectral 
weight which leaks across the Fermi level. The signature 
of such a pseudogapped Fermi surface is also observed in 
TaS 2 and TaSe2 pj| as well as in high T c superconduc- 
tors and has remained a controversial topic. 
Comparison of the experimental data with the theoretical 
DFT Fermi surface map shown in Fig. (St) for the undis- 
torted trigonal structure shows that the symmetry of the 
undistorted Fermi surface is clearly dominant. In con- 
trast to the superspots in LEED data, no clear evidence 
for the superstructure in the form of additional back- 
folded features is apparent. Voit et al |4(| have argued 
that in the presence of a weak superimposed periodic po- 
tential, the spectral weight remains on the unperturbed 
bands. An additional potential due to the (3x1) super- 
structure manifests itself through the opening of small 



gaps localized at the new Brillouin zone borders. In the 
presence of three domains, details of such a gap structure 
are buried below the bands of the other two domains, 
which do not experience the potential in this direction, 
and unfortunately it is impossible to deconvolute the con- 
tributions from the various domains. 
It should be noted that DFT for the undistorted com- 
pound predicts a truly metallic Fermi surface, whereas 
our experimental ARPES data of the distorted structure 
does not display any quasiparticle crossings. This indi- 
cates that the transition from the high symmetry struc- 
ture to the monoclinic structure is driven by a gain in 
electronic energy all over the Brillouin zone. The re- 
moval of the entire Fermi surface is not consistent with 
an explanation based solely on a 2D Peierls scenario. 
Additional information on the occupied and empty state 
electronic structure of NbTe 2 is obtained via STM. The 
chain like structure observed by STM within the domains 
(Fig. b)) is a consequence of the anisotropy of the 
(3x1) lattice and indicates a possible nesting scenario 
as the origin of the distortion (see later). For compar- 
ison between tunneling and photoemission spectra, we 
have pasted the peak positions determined from the STS 
spectrum Fig. QJ) onto the ARPES spectra Fig. QJi). 
Although it is difficult to tell, which part of /c-space is 
sampled by STS, features A, B, C, D and E can be clearly 
identified with their counterpart in the ARPES data. As 
already concluded from ARPES data, the STS spectrum 
does not exhibit a clear gap at the Fermi level. In the 
pseudogapped region between feature E and G, several 
small peak shoulders such as F are observed, indicating 
a finite density of states. The theoretical bandstructure 
discussed in the next section will allow further interpre- 
tation of these data. 

The absence of any clear quasiparticle crossing stands 
in contradiction with the metallic character of the resis- 
tivity vs temperature curves 9] and can, as discussed 
above, not be explained by only taking into account the 
Peierls scenario. Electron-phonon coupling is expected to 
be relatively strong for the tellurides 6] , consequently po- 
laronic effects may play a role. Polarons recently received 
increased attention for the interpretation of anomalously 
broad ARPES features IS EI El. Within the Fermi- 
liquid picture, ARPES peaks are attributed to quasipar- 
ticle excitations, whose lifetime increases when approach- 
ing the Fermi level. This procedure is well justified by 
recent theoretical calculations of the spectral function for 
the spinless Holstein model in the weak coupling regime 
|4J,|45(. In contrast in the strong coupling regime, the co- 
herent quasiparticle band flattens considerably and pos- 
sesses exponentially small spectral weight 0, EH • Most 
spectral weight is transferred to a broad incoherent part 
on the high energy side. Whereas the center of mass or 
first moment of the spectral function remains unaffected 
by the electron-phonon interaction at low band- fillings, 
it experiences a rigid shift proportional to the polaron 
binding energy at higher fillings |46|. A polaronic sce- 
nario would thus allow to explain the apparent absence of 
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FIG. 4: Comparison between ARPES and STS data, a) Room temperature ARPES spectra along FM. b) STM topography 
of the (3 x 1) phase of NbTe2 at 77 K (V=3 mV, 1=2.9 nA). c) STS spectrum at 77 K measured with a lock-in amplifier, 
modulation frequency 1 kHz, modulation amplitude 30 mV. The sample-tip distance was chosen in order to have a tunneling 
current of 0.8 nA for a bias voltage of -1.5 V. 



quasiparticle crossings as well as the broadened line shape 
in the experimental ARPES spectra of NbTe2- However, 
an ideal experimental model system suited for ARPES 
measurements, for which the theoretical predictions for 
polaronic spectral signatures can be verified, still has to 
be found. 



C. Electronic bandstructure 

Comparing NbTe2 and TaTe2 to their homologues in 
the sulfide and selenide family, such as TaS2, TaSe2, NbS2 
and NbSe2 , it is tempting to interpret the monoclinic dis- 
tortion in terms of a CDW phase. The existence of such 
a phase requires electron-phonon communication in the 
undistorted compound. Trigonal 7T-NbTe2 is not avail- 
able for experiment. In order to make some progress, we 
investigated the electronic structure of the undistorted 
crystal in the framework of DFT (see Appendix D for struc- 
tural details). 

In Fig. we show the bandstructure of 7T-NbTe2 

along high symmetry directions obtained with the 



APW+lo basis set. The positions of the high symme- 
try points in the Brillouin zone are indicated in Fig. |2Ji). 
The overall agreement with the bandstructure obtained 
using the pseudopotential method (not shown) is very 
good. Slight variations can be attributed to the different 
exchange-correlation functional and convergence param- 
eters. 

From a simple ionic picture, one would expect the Nb 4+ 
ions to have only one remaining d electron, resulting in 
a half-filled band crossing the Fermi level, while the six 
Te 5p bands from the two Te atoms are fully occupied. 
Obviously this picture neglects all other bonding effects, 
since an appreciable admixture of Nb Ad states exists in 
the Te p bands, indicating covalent interactions and less 
ionic bonding. Integration of the partial charges inside 
the muffin tin sphere of the respective atom allows to 
identify the character of the different bands and correctly 
reproduces this coarse prediction. The first four empty 
bands above the Fermi level are the remaining four Nb 
Ad bands. 

Due to the octahedral coordination of the Nb atoms be- 
tween the Te atoms and the resulting crystal field, the 
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FIG. 5: a) APW+lo bandstructure along high symmetry 
directions for iT-NbTe2. b) Cuts through the Fermi surface 
at k z = 0, k z = c*/4, k z = c*/2. Nesting vectors of length 
q = l/3a* are indicated. 



five Nb d bands are split into a lower triplet of ti g states 
and an upper doublet of e g states separated by a small 
gap at 2 eV binding energy. The a bonding e g orbitals 
have higher energies because they interact strongly with 
the neighboring Te atoms. The orbital degeneracy of the 
octahedral t2 g manifold is reduced in a Jahn- Teller like 
fashion by a trigonal elongation of the Te octahedra along 
the c axis. 

The P3ml space group contains an unique z axis per- 
pendicular to the layers of the crystal. Further insights 
can be obtained by dividing the d orbitals into (a) d z i 
(out-of-plane orientation), (b) d x i_ y i, d xy (in-plane ori- 
entation) and (c) d xz , d yz . The half- filled, lowest-lying 
tig band, which crosses the Fermi level, has mainly Nb 
d z 2 character, whereas the remaining two bands of the 
tig manifold (first two unoccupied bands) exhibit domi- 
nant Nb d x 2_ y 2 and d xy character. While these in-plane 
d orbitals do not interact strongly with the Te p orbitals, 
the d z 2 orbitals, due to their orientation towards the Te 
layers, are more strongly hybridized with the Te orbitals 



especially around the Brillouin zone center. This orbital 
resonance might be at the origin of the buckling of the 
Te atoms. Whereas the formation of 'trimers' by the Nb 
atoms is consistent with a Peierls scenario, the buckling 
of the Te layer rather points towards a band Jahn- Teller 
distortion [47j . However, inspection of Fig. ^ shows, 
that the Te atoms of type c which fall in between the 
Nb 'trimers' are shifted towards the Nb layer, whereas 
the other two Te atoms, labelled a and b, remain ap- 
proximately at the original distance. Since the 'trimer- 
ization' of the Nb atoms reduces the overlap between the 
Te atoms of type c[I]and its neighboring Nb atoms, these 
Te atoms approach the Nb layer to recover the overlap. 
We now compare the theoretical bandstructure for the 
undistorted structure with ARPES and STS data in Fig. 
01 from the distorted crystal. We identify feature G on the 
unoccupied side of the STS spectrum with the ti g dou- 
blet which disperses around 1 eV. These empty bands 
appear to be only weakly affected by the reduction of 
the symmetry towards the monoclinic space group. Due 
to the hybridization of the d z 2 band with the Te p bands 
and the modification of the bandstructure induced by the 
distortion to the monoclinic structure, an identification 
of the features in the occupied part of the spectrum is 
not so straightforward. The theoretical dispersion along 
TM is qualitatively in agreement with the ARPES data, 
where most bands have the tendency to disperse towards 
higher binding energies when going from f to M. How- 
ever, whereas the Nb d z 2 band clearly crosses the Fermi 
level in the theoretical bandstructure, no such crossing is 
observed in our experimental results. This indicates that 
the distortion to the monoclinic structure has profound 
effects on the Nb d z 2 band. 

We also computed the bandstructure of NbTe2 in the 
monoclinic structure (not shown). The distortion re- 
duces the density of states at the Fermi level approxi- 
mately by a factor of 2. However, the pseudogap observed 
above the Fermi level in the STS spectrum (Fig. is 
not reproduced by theory, since a considerable amount 
of states is located in this region. This discrepancy is 
possibly caused by polaronic effects as discussed above. 
Furthermore localization of the electrons decreases the 
bandwidth W of the d z 2 band and consequently the cru- 
cial parameter W/U with U the on-site Coulomb repul- 
sion energy. This opens up the possibility of increased 
correlation effects. Mott scenarios have been suggested 
for 7r-TaS 2 48] and 7r-TaSe 2 0. However, instead 
of trying to explain the discrepancy between experiment 
and theory with the insufficient treatment of correlation 
effects within LDA or GGA, we wish to point out that 
the disagreement between theory and experiment may 
be caused by the theoretical treatment of the distorted 
structure within DFT or more generally within the Bloch 
theory of periodic crystals. Since the monoclinic cell 
contains 18 inequivalent atom positions, we obtain six 
times more bands than for the bandstructure calculated 
for the trigonal unit cell. The huge amount of bands ren- 
ders the comparison with the experimental data difficult. 
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In several studies on distorted compounds, we noticed 
that the spectral weight observed in ARPES experiments 
tends to exhibit the symmetry of the undistorted struc- 
ture and that backfolded features carry generally only a 
small amount of spectral weight. The rigorous backfold- 
ing of bands within the theoretical description does not 
appear to apply to real systems. A first step towards 
a theoretical framework taking into account this obser- 
vation has been taken by Voit et al. 40], who weights 
the eigenvalues of the distorted structure obtained from 
a simple tight-binding model by the projection of the 
corresponding eigenvectors onto the eigenvectors of the 
undistorted structure. For comparison between theoret- 
ical and experimental data, it would be clearly desirable 
to implement an equivalent scheme into ab initio codes. 
When two portions of the Fermi surface are flat and par- 
allel, nesting occurs, and the susceptibility diverges loga- 
rithmically. In the isotropic electron gas, favorable nest- 
ing conditions are only encountered in one dimension, 
where the Fermi surface consists of two points. However, 
materials with an anisotropic Fermi surfaces may exhibit 
regions where scattering becomes more singular than in 
the isotropic electron gas. The presence of the (3x1) 
superstructure in the monoclinically deformed structure 
would then imply a nesting vector q = |a*. Figure Eb) 
presents three horizontal cross sections through the Fermi 
surface of 1 T-NbTe 2 for k z = |kj_| = 0, \c* and \c* . The 
black areas indicate occupied, the white areas unoccupied 
states of the Nb d z i band. Based on a Fermi surface ob- 
tained by extrapolation from the calculated results for 
the IT sulphides and selenides, Wilson 6] already noted 
that a wave vector of q = |a* is to large for nesting 
across M and M\ However, since NbTe 2 is not an ideal 
2D crystal, nesting becomes possible across L and L' as 
shown in Fig. Et>). The arrows indicate one family of the 
three symmetry-related experimental Fermi surface nest- 
ing vectors q = |a*. Furthermore the nested areas are 
relatively large encouraging a Fermi surface nesting sce- 
nario. Obviously, the contributions from different nesting 
vectors to the static susceptibility are hard to estimate 
on the basis of Fermi surface cross sections. In order to 
obtain quantitative confirmation for the q = |a* nesting 
vector, we integrated the DFT bandstructure to obtain 
the susceptibility for all vectors within the Brillouin zone. 

D. RPA susceptibility 

For the computation of the static susceptibility, the 
following expression has been used ji^ : 

X(q) = ^( e n',k+q-en,k). (1) 

n',n,k 

The Dirac S gives a contribution of either 1 or depend- 
ing on whether q is a nesting vector or not. Matrix 
elements are neglected, thus all electron-hole pairs are 
treated on an equal basis. 

The results of our calculation are presented as linear gray 




FIG. 6: RPA susceptibility in the first Brillouin zone at 
q z = (TMK plane), \c and ~c* (ALH plane) obtained by 
integration of the APW+lo Fermi surface of Fig. EJd). The 
peaks marked la and lb occur exactly at q x = |a*. Feature 
2 is close to the nesting vector of the vT9 x y/l9 CDW phase. 



scale plots in Fig. ED with white indicating a large re- 
sponse of the electron system. Strong nesting is present 
for small, but non-vanishing q vectors. These contribu- 
tions are due to intraband contributions from weakly dis- 
persing bands and can be reduced by choosing a smaller 
energy window. 

Highly interesting is the peak at q = |a* along the TM 
and rM' directions in Fig. (feature la). We associate 
this peak with a nesting vector leading to the (3x1) 
superstructure observed by LEED. Thus, the electronic 
structure of trigonal NbTe2 appears unstable with re- 
spect to a potential with wavevector q = -a*. The peak 
is not confined to the TMK plane, but is smeared out 
along q z , thus allowing an out-of-plane component for the 
nesting vector. A second maximum (feature lb) with the 
same in-plane coordinates is seen along the AL and AL' 
direction. We wish to draw attention to the fact that 
the nesting vector sketched in the ALH plane of Fig. \5jp) 
corresponds to feature la, which lies in the basal plane, 
whereas feature lb connects parts of the Fermi surface 
with different k z . Our calculation however does not re- 
produce a single peak at q = -a* + |c*, which would 
be required to explain the occurrence of the (3x1x3) 
superstructure. At present, it is unclear, if the shift be- 
tween successive layers is directly induced by the nesting 
scenario or a consequence of it. According to the calcu- 
lation, nesting takes place in the Nb d z i band and since 
the Nb atoms are screened by the surrounding Te layers, 
we may assume that the nesting mechanism operates in 
each individual Te-Nb-Te sandwich independently. As 
a consequence one might argue that the resulting CDW 
adjusts its phase in each sandwich, so as to minimize the 
repulsive inter-sandwich interaction and to maximize the 
attractive intra-sandwich energy. Inspection of Fig. 
shows, that the Te atoms closest to the Nb layer, type c, 
falls approximately in between the two Te atoms, type a 
and 6, of the next sandwich which are further away from 
the Nb atoms. This maximizes the distance between in- 
dividual Te 'anions' of successive Te-Nb-Te sandwiches 
and increases the overlap with the Nb 'cations'. In this 
framework, the tripling of the unit cell perpendicular to 
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the layers is a consequence of the in-plane (3x1) re- 
construction associated with feature la, which in turn 
is triggered by Fermi surface nesting. This would also 
explain, why we do not observe a (3 x 1 x 2) reconstruc- 
tion associated with feature lb, since such a configuration 
does not minimize the inter-sandwich interaction. 
A triple-axis distortion as in TaS 2 MM, 

in which sur- 
rounding metal atoms are shifted radially and within the 
plane towards a central metal atom to form a contracted 
star (see 16] for a sketch. flTlH^ . is reserved to clusters 
of 6n+l metal atoms, where n is the number of shells sur- 
rounding the central atom. This leads to a very precise 
condition on the nesting vector q = 1/ V6n + la* . Since 
the Fermi surface of NbTe2 exhibits dominant nesting at 
q = | a*, such a scenario is excluded, and the crystal 
locally selects one of the directions, which leads to the 
chain like (3x1) distortion and the breakup into do- 
mains. 

A second maximum (feature 2) at q = 0.19a* along TK 
might account for the star-like (a/19 x \/l9) CDW phase, 
where three shells (n — 3), each containing six metal 
atoms, are shifted towards a central atom, although it is 
slightly displaced from the ideal value q = l/^/Vda* = 
0.23a*. This nesting vector leads to an incommensurate 
phase, which was observed after cooling of heat-pulsed 
crystals to a temperature just above room temperature 
0. Our q = 0.19a* agrees with Wilson's proposal for 
nesting across the M and M' points. The absence of an 
out-of-plane component of this nesting vector in our cal- 
culation is confirmed by experiment 6] . Upon cooling to 
room temperature, the (y/l9 x \/l9) CDW rotates away 
from TK by 6.6° 6] to become commensurate with the 
parent lattice. Such a second-order incommensurate-to- 
commensurate (lock-in) phase transition has been mod- 
elled theoretically via a Landau free energy expansion 
[5l|, describing the competition of the terms that de- 
termine the individual periodicities and the term that 
promotes commensurability via gap formation. A sim- 
ilar scenario is followed by the (Vl3 x VT3) CDW of 
7T-TaS2, where in contrast to NbTe2 the nesting vector 
points along TM. 



E. Phonon bandstructure 

The occurance of a maximum in the electron suscepti- 
bility alone does not explain the distortion to the mon- 
oclinic structure. The presence of a perturbation with 
the corresponding q vector is necessary. In the one- 
dimensional Peierls scenario this potential is provided by 
a soft phonon mode. 

The DFPT phonon bandstructure for the relaxed trigo- 
nal NbTe2 structure obtained by diagonalization of the 
dynamical matrix along high symmetry lines is shown 
in Fig. [7| The lowest lying acoustic branch exhibits 
imaginary frequencies. DFPT contains the implicit as- 
sumption that phonons are simple harmonic modes. Soft 
modes are by definition anharmonic and their frequency 




r MK r A LH A 

FIG. 7: Phonon dispersion of iT-NbTe2 along high sym- 
metry lines obtained by the response function capabilities 
of ABINIT. Frequencies below meV are imaginary. All 
free degrees within the trigonal P3ml space group were re- 
laxed using the Broyden-Fletcher-Goldfarb-Shanno minimiza- 
tion scheme as implemented in the ABINIT code. The opti- 
mized LDA lattice parameters underestimate the values de- 
rived from Brown's experimental values 3] for the trigonal 
structure by less then 1 %, following the general trend ob- 
served for LDA results 52]. The LDA equilibrium value for 
Zred — 0.274 is also in good agreement with the derived aver- 
aged value Zred = 0.277 (see Appendix,). 



goes to zero. Zero frequency implies that the lattice 
structure is unstable and will transform, typically, to a 
lower symmetry phase. In the extreme case, electronic 
structure calculations may give an imaginary phonon fre- 
quency indicating that the ideal structure is unstable 
(53|. The phonon frequencies are the square roots of the 
eigenvalues of the dynamical matrix. Imaginary frequen- 
cies correspond to negative eigenvalues of the dynamical 
matrix. A negative entry in the diagonalized dynamical 
matrix contributes a negative energy to the total Hamil- 
tonian, indicating that the expansion was not carried out 
around the equilibrium configuration. Thus there exists 
an energetically more favorable configuration. At high 
temperature, the lattice has sufficient energy to overcome 
the energy barrier between two or more symmetry-related 
variants of the low temperature structure such that the 
average observed structure has higher symmetry. In such 
cases the ideal structure is stabilized by high temperature 
and will undergo a phase transition on cooling, to a low 
temperature phase whose symmetry differs by the sym- 
metry of the imaginary mode. 

The most unstable modes in Fig. [7| occur along q = 
(1/3, 0, g 2 )a*. This strongly supports the Fermi surface 
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nesting scenario for NbTe2- Furthermore, from an anal- 
ysis of the eigenvectors of the dynamical matrix, the dis- 
torted structure may be qualitatively constructed. The 
Nb atoms oscillate predominantly within the basal plane, 
the Te atoms have a dominant out of plane component, 
which is of opposite sign for the two inequivalent Te 
atoms. 

The Raman spectrum of the monoclinic NbTe2 has been 
measured by Erdogan and Kirby at T=80 K, 300 K, 
and 420 K. No phase transition was observed in this tem- 
perature range. They identified 11 peaks in their spec- 
tra (see Tab. UJ), instead of the two Raman active modes 
predicted by group theory for the undistorted compound. 
Thus unlike in ARPES measurements, where the unre- 
constructed (lxl) structure dominates, phonon bands 
get backfolded. 

A comparison between the experimental Raman peaks 
and the LDA results is shown in Tab. [I] From a strictly 
two dimensional point of view neglecting interlayer ef- 
fects, q = | a* modes are expected to lie at the new Bril- 
louin zone center. For comparison with the experimental 
data, these modes are included in Tab. [I] A low intensity 
mode was measured at 31.5 meV and may be identified 
with the q = mode at 32.6 meV. The low intensity ap- 
pears to be reminiscent of its IR character in the undis- 
torted structure. A second peak is found in the experi- 
mental spectrum at 27.2 meV and may correspond to the 
backfolded 28.3 meV mode at q = |a*. Nine experimen- 
tal peaks are found in the range between 6.9 meV to 21 
meV and correspond to the backfolded acoustic and low- 
lying optical bands. Including backfolded LDA modes 
from q = |a*, ab initio results indicate the presence of 
a maximum of 10 modes in the range between 8.8 and 
22.4 meV. An additional, even lower lying acoustic mode 
is expected due to the stabilization of the unstable high 
symmetry mode within the monoclinic structure. With 
a rms of relative deviation of 5.8% between experiment 
and LDA, the quantitative agreement can be qualified as 
fairly good |55j |. However, not all the LDA modes are 
expected to be Raman active. Experimental data for the 
IR modes are not available in the literature. 



IV. COMPARISON WITH TaTe 2 

In this article we have concentrated on NbTe2- We 
want to stress, however, that most of our conclusions ap- 
pear to be valid as well for the isostructural TaTe2- Our 
experimental LEED and ARPES data of TaTe2 show a 
very similar behavior. Furthermore the electronic band- 
structure for trigonal TaTe2 differs only slightly from the 
one obtained for NbTe2 . We thus conclude that the nest- 
ing behavior of the Fermi surface for TaTe2 is the same 
as for NbTe2- However, TaTe2 in contrast to NbTe2 does 
not become superconducting at low temperature. 
Differences between the two compounds are expected in 
the vibrational dynamics, since the phonon frequencies 
scale as the inverse square root of the mass of the os- 



TABLE I: Experimental and theoretical Raman modes 
in meV. Experimental resolution 0.5 meV. The theoretical 
optical modes at the Brillouin zone center are labelled by 
their corresponding irreducible representation obtained from 
a symmetry analysis carried out by ABINIT. Only the even 
modes (subscript g) are Raman active. The subscript u labels 
IR active odd modes. For comparison with experiment we 
also list the LDA energies of the modes at q = |a* which 
are expected to lie on the new Brillouin zone center of the 
distorted structure. 



Experiment 


LDA 


backfolded LDA 




q = 


q = (0.34,0,0)a* 


6.9 




6.4 a 


8.7 




8.8 


10.4 




9.7 


13.0 


13.9 (Eg) 




13.6 




14.7 


16.2 




17.5,17.6 


18.5 




17.9 


19.6 


20.1 (A lg ) 




21.0 


22.4 (E u ) 


22.2 


27.2 




28.3 


31.5 


32.6 (A lu ) 





a Linearly interpolated between 4.5 meV at q = 0.2a* and 8.8 
meV at q = 0.5a*. 



dilating atoms. The acoustic branches of the phonon 
bandstructure for trigonal TaTe2 however exhibit a very 
similar topology as for NbTe2 with unstable modes along 
q = (1/3, 0, q z )a* in the lowest lying branch. In contrast 
the bands of the optical manifold are drastically rear- 
ranged. The energy at T of the highest lying A\ u mode 
is reduced to 27.6 meV for TaTe2 (32.6 meV for NbTe2). 
The E u mode changes from 22.4 meV for NbTe 2 to 19.0 
meV for TaTe2 and comes to lie between the two even 
modes E g and Ai g , whose energies at T fall within less 
than 1 meV onto the energies of the NbTe2 branches. 
It is interesting to note that the odd (subscript u) IR 
modes include Nb/Ta atom motion, whereas for the even 
(subscript g) Raman modes the Nb/Ta atoms are at rest. 
This correlates well with the rescaling of the odd branches 
and the invariance of the even branches upon substitution 
of Nb by the heavier Ta atoms. These differences in the 
vibrational spectrum of the two compounds might possi- 
bly explain why TaTe2 is not superconducting, whereas 
NbTe2 becomes superconducting below T=0.5-0.74 K. 

V. SUMMARY AND CONCLUSION 

We investigated the origins of the CDW phases of 
NbTe2- LEED experiments revealed the presence of three 
coexisting domains exhibiting a (3 x 1) superstructure, 
consistent with the structure derived from X-ray diffrac- 
tion data, TEM and STM images. We carried out a 
detailed ab initio study of the nesting properties of the 
Fermi surface of the undistorted compound and found 
a singularity in the RPA susceptibility at q = |a*. In 
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order to consolidate the absence of an out-of-plane com- 
ponent of this theoretical nesting vector with the actual 
(3x1x3) structure, we suggest that the CDW within 
each individual Te-Nb-Te sandwich adjusts its phase, so 
as to minimize the repulsive inter-sandwich interaction 
(by maximizing the distance between Te 'anions' of suc- 
cessive sandwiches) and to maximize the intra-sandwich 
interaction (by maximizing the overlap of the Te orbitals 
with their neighboring Nb orbitals). A second peak at 
q « -i=a* along TK accounts for the (\/l9 x y/l9) CDW 
phase observed by TEM on heat pulsed crystals. Ab ini- 
tio phonon calculations and a soft mode analysis support 
the Fermi surface nesting scenario. 

Using angle-resolved photoemission in the Fermi surface 
mapping mode at room temperature and at T<20 K, 
we found no quasiparticle crossings in the (3x1) CDW 
phase of NbTe2 . No phase transition was observed within 
this temperature range. Localized gaps at the Fermi 
level expected for the nesting scenario could not be ob- 
served, since the photoelectron signal is a superposition 
of three domains. Instead our ARPES spectra indicate 
a pseudogap-like signature over the entire sampled por- 
tion of the Brillouin zone, which can not be understood 
by considering only the Peierls scenario and is not repro- 
duced by DFT. The angular distribution of the spectral 
weight observed at the Fermi level is dominated by the 
residual (lxl) symmetry and resembles the metallic DFT 
Fermi surface of the undistorted compound. STS spec- 
tra indicate that the unoccupied bands are only weakly 
affected by the distortion towards the monoclinic struc- 
ture, whereas most spectral weight of the Nb d z i band is 
transferred to states below the Fermi energy. The pres- 
ence of polarons within the Peierls distorted state possi- 
bly accounts for the absence of any apparent quasiparticle 
crossing and the anomalously broad features observed in 
the ARPES spectra. 

VI. ACKNOWLEDGMENTS 



the undistorted trigonal structure, the average values in 
table llll| have been used. The monoclinic cell parameters, 
marked with a m subscript are related to the undistorted 
cell parameters, without subscript, by 



a m « 3V3a, bm ~ a, c m ~ - — 3 (A.l) 

sin p 

For a the value of b m has been chosen. The value of c 
has been obtained via equation IA.1I The average value 
of the z coordinate for Te was estimated by averaging 
z re d of Tei, Te2, Te3 in the monoclinic structure taking 
into account a small offset induced by the differences in 
z re d between Nbi and Nb2 as well as the multiplicity n 
of each atom. 



Zred(Te) = ^_^^n(re i )z, ed (re i ) (A.2) 



where N(Nb) = 6 and N(Te) = 12 is the total num- 
ber of Nb and Te atoms per unit cell respectively and 
ZrediNh) = 1. 



TABLE II: NbTe 2 structure data: a m = 19.39 A, b m = 3.642 
A, cm = 9.375 A, p = 134.58°, space group 12 (C2/m) 0. 



Atoms 


Point set 




Yred 




Nbi 


2a 


0.0000 


0.000 


0.0000 


Nb 2 


4i 


0.6397 


0.000 


0.9882 


Tei 


4i 


0.6497 


0.000 


0.2898 


Te 2 


4i 


0.2970 


0.000 


0.2148 


Te 3 


4i 


0.9961 


0.000 


0.3020 
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APPENDIX: 

The structure derived by Brown 3] is given in Tab. 
HJ Asa starting point for theoretical calculations within 



TABLE III: Averaged trigonal lT-NbTe 2 structure data: a = 
b = b m = 3.642 A, c = 6.678 A, space group 164 (P3ml). 



Atoms 


Point set 




Yred 




Nb 


la 


0.0 


0.0 


0.0 


Te 


2d 


1/3 


2/3 


0.2767 
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